COMPUTER-BASED  SKILL  QUALIFICATION  TESTING:  DEVELOPMENT  AND  VALIDATION 


o 


Christopher  G.  Koch 
Judith  A.  Englert 
Richard  E.  Vestewig 
Honeywell  Systems  and  Research  Center 
2600  Rldgway  Pkwy.,  P.0.  Box  312,  Minneapolis,  MN  55440 

John  T.  Larson 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
5001  Elsenhower  Ave.,  Alexandria,  VA  22333 

An  alternative  to  present  SQTs  for  tactical  data  systems  Is  the  concept  of  a 
computer-based  embedded  SQT  administered  on  the  operational  system  or  simulator  to 
accomplish  standardized,  objective,  hands-on  skill  qualification  assessment.  This 
paper  describes  a  computer-based  SQT  Hands-On  Component  developed  for  TACFIRE 
Artillery  Control  Console  Operators,  MOS  13C,  using  the  PLANIT  courseware  authoring 
language.  The  SQT  complies  with  TRADOC  policy  and  procedures  for  SQT  development 
and  validation  (TRADOC  Reg  351-2,  Pam  351-2).  An  expert  tryout  fOr  content  validity 
and  a  representative-soldier  tryout  for  feasibility  of  administration  were  conducted. 
A  method  for  diagnosing  and  correcting  logic  errors  In  the  courseware  (vs.  operators' 
errors)  was  applied.  Benefits  of  computer-based  SQT  testing  Include:  maximum 
fidelity  attained  by  utilizing  the  operational  system;  control,  monitoring,  and 
scoring  performed  by  the  tactical  system  computer;  centralized  or  field  unit  test¬ 
ing  capability;  Immediate,  detailed  performance  feedback  provided;  test  results 
compatible  In  format  with  TRADOC  guidelines 
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1.0  INTRODUCTION 


1 .1  Statement  of  the  Problem 

The  Skill  Qualification  Test  (SQT)  is  a  primary  means  the  Army  uses  to  assess  soldiers' 
proficiency  In  performing  critical  tasks  of  their  military  occupational  specialties 
(MOSs).  It  contains  performance -oriented  and  criterion-referenced  tests  on  critical 
tasks  selected  from  the  soldier's  manual  for  a  duty  position.  Test  results  feedback 
is  provided  to  soldiers,  supervisors,  training  managers,  and  training/test  developers. 
The  SQT  usually  contains  a  written  Skill  Component  (SC),  a  Hands-On  Component  (HOC), 
and  a  Job  Site  Component  (JSC). 

Although  the  SQT  program  was  designed  to  maximize  hands-on  performance  testing, 
problems  of  standardization.  Inter-rater  scoring  reliability,  extensive  time  and 
resource  requirements,  training  of  test  administrators,  and  overall  administrative 
feasibility  have  attenuated  Its  success.  Much  skill  qualification  testing  has  become 
primarily  written  In  nature,  via  the  SC  or  a  written  Alternate  HOC.  The  written 
tests  allow  for  standardized  administration  and  scoring,  but  often  have  low  fidelity 
to  actual  job  performance  requirements.  The  Intended  objective  of  performance-based, 
criterion-referenced  SQT  testing  could  be  more  effectively  realized  by  a)  Increased 
fidelity  of  the  SQT  (that  Is,  more  similarity)  to  job  requirements  and  task  per¬ 
formance;  b)  reduced  reliance  on  written  components;  c)  reduced  time  between  test 
adnlnl strati on  and  test  results;  and  d)  Increased  feasibility  of  unit  level  SQT 
administration. 

1 .2  Concept  of  Computer-Based  Embedded  Testing 

The  Increasingly  extensive  use  of  the  computer  In  military  systems  offers  an  oppor¬ 
tunity  to  alleviate  present  shortcomings  comnon  to  conventional  SQTs  for  tactical 
data  and  weapons  systems.  The  concept  is  computer-based  embedded  SQT  testing  In 
which  the  computer  of  a  tactical  weapons  system  monitors,  controls,  and  scores  a 
test  on  actual  job  tasks  administered  on  that  system  Itself  (or  a  simulator).  This 
paper  describes  the  development  and  validation  of  a  computer-based  SQT  Hands-On  ? 
Component  (HOC)  for  the  TACFIRE  Artillery  Control  Console  Operator  (ACCO),  MOS  13C. 

Hhe  views  expressed  In  this  paper  are  those  of  the  authors  and  do  not  necessarily 
imply  endorsement  of  the  Department  of  the  Army  or  the  Army  Research  Institute. 


Work  described  in  this  paper  was  performed  by  Honeywell  Inc,  for  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  contract  no. 
MDA903-79-C-0386 . 


The  SQT  HOC  Is  comprised  of  courseware  written  in  a  computer-assisted  instruction 
system  and  authoring  language  called  PLANIT.  It  is  administered  in  an  embedded 
mode  on  the  TACFIRE  tactical  system  either  in  the  school  or  field  setting.  The  SQT 
complies  with  TRADOC  policy  and  procedures  for  SQT  development  and  validation 
(TRADOC  Reg  351-2,  Pam  351-2)  and  offers  many  advantages  over  conventional  HOC  test¬ 
ing  for  the  MOS  13C  duty  position. 


1.3  Background 


1.3.1  TACFIRE  System.  The  TACFIRE  system  selected  as  the  testbed  for  applying  the 
embedded  SQT  concept  is  a  computerized  TACtical  FIRE  direction  system  designed  to 
coordinate  the  command,  control,  and  communications  functions  within  and  among  all 
levels  of  the  field  artillery  system  from  corps  to  forward  observers.  TACFIRE  may 
be  conceptualized  as  an  interrelated  set  of  subscribers,  each  of  whom  has  access 
to  a  common  data  base  and  a  powerful  set  of  applications  software.  The  artillery 
control  console  (ACC)  shown  In  Figure  1  Is  the  principal  man-machine  Interface  in  the 
TACFIRE  system. 


HUTCH  PANEL  ASSEMBLY 


RECEIVE  DISPLAY  (RD)  SCREEN 


SELECTION  GRID 

figure  1 . 


NUMERIC  KEYIOARD 


COMPOSE/EDIT  DISPLAY 
(C/ED)  SCREEN 

TACFIRE  Artillery  Control  Console 


The  ACCO  is  a  grade  E6  or  E7  noncommissioned  officer  responsible  for  maintaining  and 
updating  the  TACFIRE  data  base.  His  functions  include:  entering  data  required  for 
artillery  target  intelligence,  fire  planning,  tactical  and  technical  fire  control, 
and  commander's  criteria;  controlling  communications  among  the  network  of  TACFIRE 
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subscribers;  and  Initiating  ,  processing,  and/or  terminating  artillery  fire 
missions.  Task  activities  for  which  the  ACCO  has  responsibility  Include: 

o  Receiving  and  Interpreting  Information.  The  ACCO  receives  information  from 
the  subscriber  network  on  the  receive  display  (RD)  screen.  This  informa¬ 
tion  Is  encoded  In  TACFIRE  specific  formats,  and  requires  that  the  ACCO  be 
skilled  In  Interpreting  the  information  and  acting  quickly. 

o  Composing  and  sending  messages.  The  ACCO  enters  data  into  the  system  via 
special  message  formats  on  the  compose/edit  display  (C/ED)  screen,  then 
makes  proper  switch-actions  to  Initiate  processing  of  the  data  and  trans¬ 
mission  to  designated  on-line  subscribers. 

1.3.2  PLAN IT  System.  The  embedded  SQT  for  the  TACFIRE  system  was  written  In 
Programming  LANguage  for  Interactive  Teaching  (PLANIT),  a  powerful  computer-assisted 
Instruction  system  and  courseware  autForlng  language  useful  for  preparing  and 
administering  training  and  evaluation  scenarios  via  computer  terminal.  Courseware 
Is  constructed  either  Interactively  or  off-line,  and  the  author  may  execute  and/or 
edit  a  lesson  at  any  time.  PLANIT  automatically  maintains  an  audit  trail  of  each 
student's  progress  through  a  course,  thus  relieving  the  Instructor  or  scorer  of 
tedious  record-keeping  responsibilities. 

The  PLANIT  operating  system  Is  made  machine  transportable  and  has  been  Installed  on 
computer  systems  supporting  FORTRAN,  PASCAL,  and  TACPOL  (language  used  for  TACFIRE). 
For  the  TACFIRE  AN/GYK-12  application,  enhancements  to  the  operating  system  software 
were  Implemented  to  extend  PLANIT's  capabilities  to  a  specific  hardware  system. 
PLANIT's  enhanced  language  for  TACFIRE  permits  the  lesson  author  courseware  control 
of  switch  Indicator  lights,  and  Interprets  the  operation  of  ACC  function  keys. 
Outputs  may  be  directed  to  either  of  two  screens,  to  the  electronic  line  printer 
(ELP),  or  to  the  switch  panel  assembly  (SPA).  Inputs  are  processed  from  the  C/ED 
screen,  the  alphanumeric  keyboard,  or  the  SPA.  Under  control  of  PLANIT's  enhanced 
language  for  TACFIRE,  the  TACFIRE  ACC  can  operate  exactly  as  It  would  under  control 
of  the  operational  system  software  In  tactical  mode. 

2.0  DEVELOPMENT  OF  COMPUTER-BASED  SQT 


2.1  Requirements 

The  development  of  the  HOC  offered  a  unique  challenge  for  complying  with  regula¬ 
tions  that  do  not  explicitly  address  Issues  for  computer-based  SQT  testing.  Although 
some  steps  Identified  In  TRADOC  Pam  351-2  were  not  relevant  (for  example,  training 
hands-on  test  scorers  and  assessing  Inter-rater  reliability),  and  other  additional 
developmental  steps  were  Included,  the  TRADOC  regulation  and  pamphlet  served  as 
adequate  guidance  for  the  development  process. 

2.2  Development  Process 

The  major  activities  conducted  during  development  of  the  SQT  were: 

1.  Select  tasks  for  testing. 

2.  Assign  tasks  to  SQT  components. 

3.  Develop  SQT  plan. 
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4.  Develop  test  materials. 

a.  Collect  task  performance  data. 

b.  Organize  hands-on  tests. 

c.  Program  courseware  for  hands-on  tests  on  commercial  computer. 

d.  Try  out  courseware  materials  on  TACFIRE  computer. 

e.  Revise  hands-on  tests. 

5.  Develop  support  documentation— "Manual  for  the  Administration  of  the 
Hands-On  Component"  (MAHOC) ,  SQT  Notice  hands-on  test  worksheets, 

"Data  Supplement,"  and  "Operator's  Guide  for  Preparing  SQT  Materials." 

6.  Conduct  expert  tryout. 

7.  Submit  initial  draft. 

8.  Conduct  soldier  tryout. 

Pertinent  characteristics  of  the  finalized  TACFIRE  embedded  SQT  HOC  are  summarized 
in  Table  1.  Koch,  Pine,  and  Stelnhelser  (1980)  and  Koch,  Englert,  and  Vestewlg 
(1981)  provide  details  of  the  development  process  for  the  SQT. 

TABLE  1.  CHARACTERISTICS  AND  CAPABILIITES  OF  THE 
TACFIRE  EMBEDDED  SQT  HANDS-ON  COMPONENT 


Objective 

o  Assess  qualification  of  artillery  control  console  operator  (ACCO), 

MOS  13C,  skill  levels  3  and  4. 

Conditions  of  Testing 

o  Delivered  on  operational  tactical  TACFIRE  equipment  using  TACFIRE 
AN/GYK-12  computer  or  on  TACFIRE  Training  System  (TTS). 
o  Capability  of  simultaneous  testing  of  soldiers, 
o  High  fidelity  to  soldier's  manual  (FM  6-13C3,  FM  6-13C4)  tasks, 
o  Standardized  administration  at  either  centralized  (Field  Artillery 
School)  or  field  unit  Installations. 

Test  Organization 

o  HOC  composed  of  hands-on  tests. 

o  Automatic  presentation  of  all  necessary  test  Instructions  and  hands-on 
test  conditions,  standards ,  and  performance  measures. 

Test  Scoring 

o  Scoring  performed  automatically  by  TACFIRE  computer, 
o  Each  hands-on  test  scored  "GO"  or  "NO-GO"  based  on  number  of  per¬ 
formance  measures  passed  and  satisfaction  of  time  limit  standard, 
o  qualification  on  SQT  based  on  criterion  percentage  of  hands-on  tests 
scored  "GO." 

Test  Results  Feedback 

o  Inmedlate  "test  results  and  performance  feedback. 

o  Performance  feedback  specific  to  Items  at  the  performance  measure  level, 
o  Individual  Soldier's  HOC  Report  providing  tabulation  of  results, 
o  HOC  Record  Sheet  providing  official  record  of  test  results. 


3.0  VALIDATION  OF  SQT  HANDS-ON  COMPONENT 

3.1  Validation  Plan 


The  validation  phase  of  the  SQT  development  Involved  the  production  of  a  validatlo 
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(TRADOC  Pam  351-2).  The  objective  of  the  validation  was  to  examine  the  HOC  for 
content  validity,  accuracy,  perceived  fairness,  and  administration  feasibility. 
The  strategy  for  validation  of  the  HOC  Is  reflected  in  the  flow-chart  in  Figure  2. 


Two  unique  Issues  were  Important  In  assuring  an  acceptable  validation  of  the 
computer-based,  embedded  SQT.  The  first  concerned  a  necessary  distinction  between 
logic  errors  In  the  hands-on  test  courseware  and  unfair  or  inaccurate  test  content 
per  se.  Thus,  two  types  of  validation  were  required— one  for  courseware  materials 
and  one  for  hands-on  test  technical  content.  The  second  Issue  concerned  the 
requirements  for  edmlnl  strati  on  (using  the  resource;;  of  a  typical  unit)  and 
maintainability  associated  with  a  computer-based  SQT. 


EXVfRT  TRYOUT  RRLDIIR  TRYOUT 


Figure  2.  Flow  Diagram  of  SQT  HOC  Validation  Process 
3.2  Expert  Tryout 

3.2.1  Objectives.  The  objectives  of  expert  tryout  of  the  T  AC  FI  RE  SQT  HOC  were  to: 

o  Provide  TACF1RE  subject  matter  experts  an  opportunity  to  systematically 
review  each  hands-on  test  for  accuracy  of  the  task  procedures  as  defined 
by  the  soldier's  manual. 

o  Examine  the  draft  MAHOC  for  content,  readability,  and  clarity. 

o  Identify  and  Implement  any  necessary  revisions  to  the  hands-on  tests, 
Instructions,  and  draft  MAHOC. 

o  Determine  test  administration  requirements,  including  time  to  set  up  and 
administer  the  hands-on  tests. 

3-2.2  Procedure.  The  participants  in  expert  tryout  were  14  instructors  In  the 
TACFIRE  Advanced  Training  Program  at  the  U.S.  Army  Field  Artillery  School,  Fort 
Sill,  Oklahoma.  Both  noncommissioned  and  commissioned  officers  were  included 
In  the  sample.  Five  experts  completed  the  SQT3  HOC  (the  HOC  for  skill  level  3), 
and  six  experts  completed  the  SQT4  HOC  (the  HOC  for  skill  level  4).  Each  of  the 
experts  served  both  as  test  administrator  and  examinee.  A  two-hour  workshop  was 
conducted  to  familiarize  the  participants  with  procedures  for  preparing  the 
computer-based  HOC  to  be  administered. 


During  expert  tryout,  the  participants  executed  the  support  procedures  routine  to 
prepare  the  HOC  to  be  administered  and  then  performed  each  of  the  hands-on  tests 
in  the  same  manner  as  an  examinee  would.  Data  of  various  types  were  collected  to 
meet  the  stated  objectives.  Table  2  summarizes  the  type,  purpose,  and  collection 
time  of  these  data. 


TABLE  2.  EXPERT  TRYOUT  DATA  COLLECTION 


Type 

Purpose 

When  Collected 

Expert  Review  Form 

Examine  task  domain 
sampling,  hands-on  test 
accuracy,  test  instructions, 
and  test  fairness 

End  of  each  hands-on  test 

Draft  MAHOC  Review 
Form 

Examine  Draft  MAHOC 

End  of  workshop  to  train 
test  administrators 

Support  Procedures 
Review  Form 

Exaaine  test  administrator's 
support  procedures 

End  of  test  session 

General  Computer 
Instructions  Review 
Form 

Examine  general  HOC 
instructions  to  the  examinee 

End  of  test  session 

Error  Data 

Examine  hands-on  tests 

Recorded  automatically 
by  computer  during  test 
session 

3.2.3  Results.  Responses  to  review  fores  on  the  draft  MAHOC,  support  procedures, 
and  general  Instructions  Identified  areas  for  revision.  The  section  In  the  MAHOC 
dealing  with  computer  malfunction  recovery  was  considered  Inadequate,  and  certain 
portions  of  the  general  Instructions  were  rated  as  unclear.  Prerequisite  data  pro¬ 
vided  on  the  ELP  for  some  hands-on  tests  were  difficult  to  read  due  to  poor  printer 
type  element  quality  and  caused  some  undeserved  errors.  Responses  on  the  expert 
review  forms  dealing  with  technical  content  of  each  hands-on  test  Identified  problems 
In  these  areas:  task  requirements  unclear;  only  one  try  allowed  for  selecting 
message  format  switches;  optional  data  entrles/blanks  not  allowed;  alternate  legal 
forms  of  data  entry  not  allowed;  answer  match  lines  containing  typographical  errors. 

Analysis  of  experts'  errors  on  the  hands-on  tests  yielded  three  Identifiable 
causes  of  performance  measure  failures: 

o  Operator  error— Incorrect  switch-action  or  failure  to  complete  hands-on 
test  within  appropriate  time  limit. 

o  Courseware  logic  error- -ml stakes  in  the  PLANJT  courseware  that  may  cause 
unwarranted  performance  measure  failures. 

o  Inappropriate  time  standard— Insufficient  time  allowed  for  completing 
hands-on  tests. 

Distinguishing  operator  errors  from  logic  errors  required  an  examination  of  the 
printer  output  for  inconsistencies  between  the  expert's  response  and  his  detailed 
test  performance  feedback. 

The  remainder  of  the  error  analysis  focused  on  the  operator  errors.  Four  classifi¬ 
cations  of  operator  errors  and  the  average  frequency  with  which  they  were  committed 
are:  a)  incorrect  data  entry— 52% ;  b)  Incorrect  message  format  selectlon—23%; 
c)  Incorrect  switch  action— 13%;  d)  excessive  time  to  complete  the  task— 12%. 


3.2.4  Revisions .  Revisions  to  the  hands-on  test  materials  based  on  expert  tryout 
data  Included:  expanding  the  computer  malfunction  recovery  section  of  the  MAHOC; 
developing  a  "Data  Supplement"  containing  hardcopy  of  prerequisite  data  for  all 
hands-on  tests;  rewriting  portions  of  general  Instructions  in  simpler  language; 
clarifying  task  requirements;  Implementing  two  tries  to  select  correct  message 
format  switches;  allowing  optional  data  entries/blanks ;  allowing  alternate  legal 
forms  of  data  entries;  correcting  typographical  errors  In  answer  match  lines; 
correcting  courseware  logic  errors;  revising  inappropriate  time  limit  standards. 
After  modifications  to  the  SQT  HOC  had  been  Implemented,  the  hands-on  tests  were 
again  administered  to  a  subset  of  the  expert  tryout  participants  to  verify  accuracy 
of  the  revisions. 

3.3  Soldier  Tryout 

3.3.1  Objectl ves.  The  objectives  of  soldier  tryout  of  the  TACFIRE  SQT  HOC  were  to: 

o  Determine  the  HOC's  administrative  feasibility  In  a  realistic  test 
setting  with  representative  soldiers  In  the  MOS. 

o  Continue  to  examine  the  hands-on  tests  for  accuracy,  appropriateness, 
and  fairness. 

o  Perform  statistical  analyses  of  hands-on  test  performance  data. 

o  Continue  to  examine  the  draft  MAHOC  for  clarity  and  completeness. 

o  Implement  revisions  to  the  hands-on  tests,  instructions,  and  draft 
MAHOC  where  necessary. 

3.3.2  Procedure.  The  soldier  tryout  participants  were  22  soldiers  In  MOS  13C 
stationed  at  Fort  Sill,  Oklahoma.  Ten  staff  sergeants  completed  SQT3  HOC  and  ten 
sergeants  first  class  completed  SQT4  HOC.  Personnel  from  the  Directorate  of 
Training  Development,  U.S.  Amy  Field  Artillery  School  served  as  test  administrators. 

The  type,  purpose,  and  collection  time  of  soldier  tryout  data  are  shown  in  Table  3. 
The  data  derived  from jthese  sources  were  analyzed  according  to  the  process  depicted 
In  Figure  3. 


TABLE  3.  SOLDIER  TRYOUT  DATA  COLLECTION 


Type 

F\irpose 

When  Collected 

Soldier  Review  Font 

Examine  hands-on  tests  for 
fairness,  accuracy,  and 
clarity  of  instructions 

End  of  test  session 

General  Computer 
Instructions 

Review  Form 

Examine  general  HOC 
instructions  to  the 
examinee 

End  of  test  session 

Error  Data 

Examine  hands-on  tests; 
perform  statistical 
evaluation  of  HOC 

Recorded  automatically  by 
computer  during  test 
session 

Timing  Data 

Determine  test 

administration  requirements; 
examine  performance 
standards  for  task 
completion  time  ; 

Recorded  automatically  by 
computer  during  test 
session 
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Figure  3.  Flow  Diagram  of  Error  Analysis  and  Test 
Revision  Process  Following  Soldier  Tryout 

3.3.3  Results.  Responses  on  the  general  instructions  review  forms  Identified  two 
portions  of  the  Instructions  that  some  soldiers  found  unclear— action  to  take  after 
a  discovered  error,  and  amount  of  help  expected  from  the  test  site  manager. 

Responses  on  the  soldier  review  forms  evidenced  that  a  large  percentage  of  the 
soldiers  viewed  the  SQT  HOC  as  a  fair  and  acceptable  measure  of  their  ability  to 
perform  job  duties. 

Errors  committed  on  the  hands-on  tests  were  again  categorized  by  source— operator 
error,  courseware  logic  error,  and  Inappropriate  time  standard.  The  type  and 
average  frequency  of  errors  attributable  to  the  operator  are:  a)  Incorrect  data 
entry— 51%;  b)  incorrect  message  format  selection— 18%;  c)  Incorrect  switch  actlon- 
21%;  d)  excessive  time  to  complete  the  task— 10%. 

Analysis  of  the  timing  data  for  the  representative  soldiers  provided  an  empirical 
basis  for  examining  the  adequacy  of  the  hands-on  test  time  limit  standards.  An 
average  of  83%  of  soldiers  taking  SQT3  HOC  finished  the  hands-on  tests  within  given 
time  limit  standards;  92%  of  those  taking  SQT4  HOC  finished  within  the  standards. 
Three  specific  criteria  were  used  to  judge  the  adequacy  of  each  hands-on  test  time 
standard:  a)  equals  approximately  double  the  average  time  required  by  representa¬ 
tive  soldiers;  b)  exceeds  the  upper  limit  of  the  range  of  task  completion  times; 
c)  equals  a  value  defined  by  training  developers  to  be  mission-related. 

3.3.4  Revisions.  Revisions  to  the  hands-on  test  materials  based  on  soldier  tryout 
data  Included:  rewriting  portions  of  general  instructions;  Including  SQT  HOC 
administration  time  guidelines  based  on  soldier  tryout  In  the  MAHOC;  correcting 
courseware  logic  errors;  adjusting  hands-on  test  time  standards. 

4.0  ADMINISTRATION  AND  MAINTENANCE  ISSUES 

4.1  Issues  In  Administering  the  SQT  HOC 

TRADOC  regulations  for  SQT  development  stress  the  Importance  of  one  feature  that 
should  characterize  an  SQT— relative  ease  of  administration  (TRADOC  Reg  351-2). 

To  meet  this  criterion,  the  SQT  should  be  easy  to  administer  by  NCOs  at  the  E6 
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level  or  higher,  be  accompanied  by  comprehensive  and  simple  guidelines  for  test 
administrators,  and  require  only  the  personnel  and  equipment  resources  available 
at  a  typical  unit.  These  became  the  objectives  for  administering  the  TACFIRE 
embedded  SQT.  Issues  Important  for  SQT  administrative  feasibility  include  qualifi¬ 
cations  of  test  administrators,  system  configuration  and  status,  test  security, 
and  disposition  of  test  results. 

With  the  MAHOC  as  a  reference,  administering  the  computer-based  SQT  is  designed  to 
be  a  turn-key  operation  for  the  test  administrator.  He  will  be  able  to  perform 
test  set-up  procedures  explained  In  the  MAHOC,  administer  the  SQT  to  soldiers,  and 
collect  the  examinee  test  results  with  no  prerequisite  computer  programming  ex¬ 
perience  or  outside  assistance. 

The  SQT  HOC  at  either  skill  level  3  or  4  can  be  administered  on  any  existing  TACFIRE 
system  configuration,  that  Is,  battalion  or  division  artillery,  at  centralized 
locations  or  field  unit  locations.  The  SQT  materials  written  In  PLANIT  courseware 
are  transportable  via  standard  TACFIRE  Input-output  medium- -magnetic  tape  cartridge. 
The  load  procedures  are  comparable  to  those  performed  by  an  ACCO  for  TACFIRE 
tactical  system  software.  In  the  event  of  external  threat  or  other  emergency,  the 
TACFIRE  system  can  be  restored  to  operational  status  immediately  by  loading  tactical 
system  software. 

Test  security  for  the  computer-based  SQT  depends  on  safeguards  against  compromise 
by  those  who  are  not  authorized  to  administer  the  SQT  and  safeguards  against  un¬ 
authorized  modifications  to  the  courseware.  The  PLANIT  software  provides  for  both 
safeguards  since  log-in  Identifications  used  by  operators  and  examinees  restrict 
their  access  to  only  those  functions  In  the  courseware  Intended  for  their  use. 

For  the  computer-based  SQT,  the  scoring  procedure  will  differ  from  the  conventional 
approach.  The  TACFIRE  computer  will  perform  the  scoring  for  each  examinee,  tabulate 
test  results,  and  print  a  complete  HOC  Record  Sheet.  This  record  sheet  can  be 
transcribed  directly  to  the  SQT  marksense  form  used  by  the  training  standards 
officer  to  compile  test  results. 

4.2  Maintainability  Issues  and  Procedures 

There  are  four  general  levels  of  maintenance  for  the  computer-based,  embedded  SQT 
considered  Important  for  an  overall  goal  cf  ease  of  maintenance.  The  levels  of 
maintenance  are  listed  in  Table  4  ordered  from  specific  to  general,  together  with 
the  primary  objective  associated  with  each. 

The  aspects  of  TACFIRE  equipment  maintenance  relevant  to  actual  system  operation 
also  affect  SQT  operation.  In  the  event  of  equipment  malfunctions,  the  testing  is 
halted  but  no  performance  data  are  lost  from  core  memory.  When  the  system  is 
restored,  PLANIT  automatically  resumes  HOC  testing  at  the  point  where  each  examinee 
was  stopped. 

Annual  revisions  to  an  SQT  mandated  by  TRADOC  are  accommodated  by  a  large  base  of 
validated  hands-on  tests.  The  SQT  developers  need  only  select  a  different  subset 
of  hands-on  tests  each  year  to  optimally  represent  extant  job  duties.  This  selection 
requires  no  modifications  to  the  courseware. 
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TABLE  4.  MAINTENANCE  OBJECTIVES 


Level  of  Maintenance 

Primary  Objective 

1. 

Maintenance  for  TACFIRE 
equipment  on  which  the 

SQT  HOC  is  administered. 

Ensure  test  reliability  by 
minimizing  equipment  failure 
and  down  time. 

2. 

Annual  revisions  to  SQT  contents 
in  accordance  with  TRADOC 
regulations. 

Provide  flexibility  to  use 
developed  and  validated  hands-on 
tests  for  more  than  one  test  perioo. 

3. 

Modifications  and  additions  to 
the  courseware  comprising 
hands-on  tests. 

Provide  capability  to  implement 
modifications  and  additions  to  the 
courseware  interactively,  easily, 
and  rapidly. 

4. 

Enhancements  and  troubleshooting 
support  for  the  software  systjm. 

Provide  quality  assurance  for 
system  software. 

New  Interface  requirements  and  operating  system  software  releases  will  drive  the 
training  and  testing  requirements  eor  TACFIRE  and  any  evolving,  complex  system. 
Supplementing  the  TACFIRE  SQT  with  Pew  or  revised  test  materials  Is  an  Important 
capability  to  accommodate  extensions  the  scope  of  the  SQT.  A  technician  with 
computer  programming  experience  has  the  qualifications  to  edit  existing  courseware 
and  develop  new  courseware  for  the  TACFIRE  embedded  SQT.  The  more  common  types  of 
courseware  modifications  that  will  be  required  are  quite  simple  and  can  be  accom¬ 
plished  Interactively  from  the  ACC  terminal  at  any  time. 

At  the  most  general  level  of  maintenance  for  the  computer-based  SQT  Is  support  of 
any  enhancements  for  the  PLANIT  operating  system  software.  The  evidence  for 
maintainability  and  supportablllty  of  the  PLANIT  software  Is  derived  from  many 
trouble-free  Installations  of  PLANIT  nationwide,  spanning  a  wide  variety  of 
applications. 


5.0  CONCLUSIONS 

Conclusions  warranted  by  the  results  of  the  computer-based  SQT  development  and 
validation  may  be  relevant  both  as  an  overview  of  the  present  work  and  as  research 
and  development  Issues  for  other  SQTs  designed  to  be  delivered  on  operational  tac¬ 
tical  data  systems.  First,  the  TACFIRE  embedded  SQT  appeo'**  to  be  enthusiastically 
accepted  by  the  user  community.  The  extensive  Involvement  of  subject  matter 
experts  and  TRADOC  personnel  during  the  development,  validation,  and  review 
process,  together  with  the  developer's  responsiveness  to  TRADOC  regulations  and 
user  Input,  are  thought  to  be  Important  contributors  to  this  acceptance. 

The  TACFIRE  application  of  computer-based,  embedded  HOC  testing  Is  feasible  and 
effective  in  fulfilling  the  Intent  of  the  Arrny  SQT  program.  Performance  of  hands-on 
tests  on  the  TACFIRE  equipment  closely  resembles  the  performance  of  actual  job 
duties.  The  TACFIRE  computer  accomplishes  objective  and  accurate  scoring  of 
hands-on  tests.  The  SQT  is  also  feasible  from  the  standpoint  of  administration 
and  maintenance,  requiring  no  programming  experience  to  administer  and  no  other 
resources  or  equipment  than  those  available  at  the  typical  field  unit.  The  PLANIT 
courseware  materials  are  structured  in  a  modular  format  and  are  easy  to  modify  and 
supplement. 
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Compliance  with  TRADOC  regulations  for  SQT  policy  and  procedures  by  the  computer-based 
SQT  does  not  differ  significantly  from  conventional  SQTs.  The  validation  plan, 
expert  tryout,  soldier  tryout,  data  analysis,  and  test  content  revisions  were  con¬ 
ducted  In  accordance  with  regulations.  The  MAHOC  closely  follows  the  TRADOC  re¬ 
commended  format.  Test  results  are  computer-generated  in  a  form  compatible  with 
TRADOC  SQT  scoring  procedures. 

PLANIT  has  been  a  successful  vehicle  for  development  of  the  TACFIRE  embedded  SQT. 
PLANIT  capabilities  and  advantages  Include:  extensive  record  keeping  of  student 
performance;  analysis  of  Incorrect  answers  to  provide  detailed  feedback;  trans¬ 
portability  of  courseware;  ease  of  courseware  modification.  The  success  of  PLANIT 
In  the  Implementation  of  the  TACFIRE  embedded  SQT  recommends  It  as  a  candidate  for 
other  developments  on  computer-based  tactical  data  systems. 


REFERENCES 

Guidelines  for  Development  of  Skill  Qualification  Tests.  TRADOC  Pamphlet  No. 
351-2  (Draft),  Department  of  the  Army,  Headquarters,  U.S.  Army  Training 
and  Doctrine  Coninand,  Fort  Monroe,  VA,  1980. 

Koch,  C.G.,  Englert,  J.A.,  &  Vestewlg,  R.E.  Development  and  Validation  of  a 
Computer-Based  Skill  Qualification  Test— Final  Report.  Report  No.  81SRC58 , 
Honeywell  Systems  and  Research  Center,  Minneapolis,  MN,  September  1981. 

Koch,  C.G.,  Pine,  S.M. ,  A  Kingsbury,  G.G.  Development  of  a  Computer-Based 
Skill  Qualification  Test— Final  Report.  'Report  &OSRC52,  Honeywell  Systems 
and  Research  Center,  Minneapolis,  MN,  July  1980. 

Koch,  C.G.,  Pine,  S.M. ,  &  Stelnhelser,  F.H.  "Development  of  a  Computer-Based 
Skill  Qualification  Test."  Proceedings  of  the  Military  Testing  Association 
22nd  Annual  Conference,  Toronto,  Ontario,  27-31  October  1$6Q. 

Skill  Qualification  Test  (SQT)  Policy  and  Procedures.  TRADOC  Regulation  No. 
351-2,  Department  of  the  Army,  Headquarters,  U.S.  Army  Training  and  Doctrine 
Command,  Fort  Monroe,  VA,  21  April  1980. 

Soldier’s  Manual ,  TACFIRE  Operations  Specialist,  MOS  13C,  Skill  Level  3. 

Field  Manual  No.  FM  6-13C3,  Headquarters,  Department  of  the  Army,  December 
1980. 

Soldier's  Manual ,  TACFIRE  Operations  Specialist,  MOS  13C,  Skill  Level  4. 

Field  Manual  Mo.  FM  6-13C4,  Headquarters,  Department  of  the  Army,  December 
1980. 


